A Textual Analysis of Every U.S. General Election Debate
Dominic Russel
10/19/2020
The first U.S. Presidential debate was widely covered as a covered …. While analysts… One way of contextualizing this debate in U.S. history is through a textual analysis of the …. This removes some important signals (the tone individuals speak with, for example), but…
To do so, I scraped and cleaned the transcripts of every general election Presidential and Vice Presidential debate in U.S. history. The final dataset includes over 13,500 questions and responses organized by debate and speaker, from the first Kennedy-Nixon debate in 1960 to Harris-Pence VP debate in 2020. You can find the data here.
Simple Sentiment Analysis
One way to analyze these data is to look at how positive or negative the candidates are in their responses — the sentiment of their arguments. At its most simple, sentiment analysis counts the number of positive words (“excellent”, “brilliant”, “win”) and negative words (“worst”, “horrible”, “fraud”) used in a text. In this analysis, a debater who uses more positive words than negative words will have a more positive average sentiment (and vice versa).
Using the AFINN dictionary, which assigns scores of -5 to 5 to a corpus of nearly 2,500 words, I calculate the average sentiment of every presidential debate candidate since 1960. It is important to note before diving into the results that sentiment analysis is an imperfect science, and that any results are dependent on certain assumptions about language. Using the same dictionary of words over time, for example, might be problematic if a word that is more emotionally charged today than in the past.
With these caveats aside, the plot below shows the average sentiment of presidential debaters from the two major parties.
Average Sentiment of U.S. Presidential Debaters (AFINN)
The first thing that jumps out, perhaps unsurprisingly, is that Trump is historically negative in his debate performances. The average sentiment of his words is more than twice as negative than even the closest comparisons (Reagan in 1980 and 1984, and Biden in 2020). By contrast, the most positive debate performances were delivered by Gerald Ford, the George Bushes in their first terms, and both John F. Kennedy and Richard Nixon in the first presidential debate. In general, sentiment appears to have declined over time between the first two years of presidential debates in 1960 and 1976 (there were no presidential debates in the 1964, 1968, and 1972 elections) and the last two in 2016 and 2020.
We can also break out sentiment by each individual debate performance. Here we also include VP debates and the three candidates outside of the two major parties that made the debate stage. The graph below shows how historically negative each of Trump’s debate performances have been.
Average Sentiment by U.S. General Election Debate Performance (AFINN)
Why are Trump’s debates rated as so negative? Because AFINN is based on this simple dictionary scoring method, we can look “under the hood” and find out. The tables below compare the most positive/negative and frequently used words for Trump and for a debator with some of the most positive scores — John F. Kennedy. The purple words are negative, arranged by sentiment value then by usage. The green words are positive, arranged similarly so the most positive, most used words are at the bottom. The farthest right column shows how many times Trump/Kennedy used each word, as a percent of every word they used in the AFINN dictionary.
| Word | Sentiment Value | Share of Neg & Pos Words (%) |
|---|---|---|
| bastards | -5 | 0.08 |
| fraud | -4 | 0.50 |
| hell | -4 | 0.34 |
| catastrophic | -4 | 0.08 |
| fraudulent | -4 | 0.08 |
| bad | -3 | 4.79 |
| lost | -3 | 1.51 |
| worse | -3 | 1.09 |
| worst | -3 | 1.01 |
| horrible | -3 | 0.92 |
| fantastic | 4 | 0.08 |
| funny | 4 | 0.08 |
| miracle | 4 | 0.08 |
| wins | 4 | 0.17 |
| amazing | 4 | 0.25 |
| winner | 4 | 0.25 |
| winning | 4 | 0.25 |
| win | 4 | 0.50 |
| wonderful | 4 | 0.67 |
| outstanding | 5 | 0.08 |
| Word | Sentiment Value | Share of Neg & Pos Words (%) |
|---|---|---|
| lost | -3 | 0.50 |
| crisis | -3 | 0.38 |
| worst | -3 | 0.38 |
| killed | -3 | 0.25 |
| withdrawal | -3 | 0.25 |
| bad | -3 | 0.13 |
| destroy | -3 | 0.13 |
| destruction | -3 | 0.13 |
| died | -3 | 0.13 |
| disastrous | -3 | 0.13 |
| popular | 3 | 0.13 |
| praised | 3 | 0.13 |
| prosperous | 3 | 0.13 |
| wealth | 3 | 0.13 |
| vitality | 3 | 0.25 |
| breakthrough | 3 | 0.38 |
| successful | 3 | 0.88 |
| fantastic | 4 | 0.13 |
| triumph | 4 | 0.13 |
| win | 4 | 0.38 |
Whereas Kennedy did not use any word with a sentiment value lower than -3, one percent of Trump’s AFINN matching word had sentiment values of -4 or lower. Also dragging him down is likely his repeated use of the word “bad”, which alone makes up five percent of his negative/positive words. On the other hand, Trump does also use more very emotionally charged positive words than Kennedy (in particular “wonderful” and variations of the word “win”). But Kennedy also did repetedly use a number of fairly positive words like “successful”, “breakthrough”, and “win”.
A final thought on these sentiment analyses: while the first plot showed that the sentiment in Biden’s first 2020 debate performance was generally quite negative compared to the average of past candidates, Biden’s individual debate performances haven’t been particularly negative. The graph below highlights Trump’s four presidential debates and Biden’s two VP debates (in 2008 and 2012) and one presidential debate.
Average Sentiment by U.S. General Election Debate Performance – Biden/Trump (AFINN)
Additional Measures of Sentiment and Emotion
One potential problem with the simple dictionary based lookups used in the previous analysisis is that they could overlook the presence of “valence shifter” words that can change sentiment. For example, if someone used the word “bad” we might want to know if they actually said something was “not bad” or “very bad”. Another thing we might be interested in is looking beyond simply a negative/positive scale, and instead at specific emotions like anger, disgust, and trust.
The sentimentr package implements solutions for both. Without going too far into the details, the package uses a similar set of dictionary methods but with an expanded vocabulary, accounting for valence shifters, and word tagging for particular emotions.
Using this package, we can first re-create our graph of debater sentiment over time for the two major parties.
Average Sentiment of U.S. Presidential Debaters (sentimentr)
While some points change, the main results hold: Trump is historically negative and Ford and the first debate Bushes are very positive.
We can also look at particular emotions…
Average Emotional Scores of U.S. Presidential Debaters (sentimentr)
Most Commonly Used Bigrams
To understand, below we present bigram word clouds…
Here are five from 1960, 1984, 2004, 2016, and 2020. You can find them for every presidential debate here.
Principal Components Analysis
One more thing we can do is PCA, using the..
We do lose some interpreitability, but essentially…
In 2004, they did debates focused entirely on one set of issues. This 2004 cloud was focused entirely on foreign policy and homeland security (was also a topic discussed at the Pence-Kaine debate).
Can’t be totally unique words (must be used in 10% of debates)
Instead we can do it only using…
Here Trump is truly in a league of his own… His supporters say he speaks in a different way…
Thanks for reading! If you enjoyed this please share it. You can find out the code for this document, as well as the cleaned data in this Github repository.